1. Field of the Invention
The present innovation relates generally to artificial neural networks and more particularly in one exemplary aspect to computer apparatus and methods for pulse-code neural network processing of sensory input.
2. Description of Related Art
Artificial spiking neural networks are frequently used to gain an understanding of biological neural networks, and for solving artificial intelligence problems. These networks typically employ a pulse-coded mechanism, which encodes information using timing of the pulses. Such pulses (also referred to as “spikes” or ‘impulses’) are short-lasting (typically on the order of 1-2 ms) discrete temporal events. Several exemplary embodiments of such encoding are described in a commonly owned and co-pending U.S. patent application Ser. No. 13/152,084 entitled APPARATUS AND METHODS FOR PULSE-CODE INVARIANT OBJECT RECOGNITION”, filed Jun. 2, 2011, and U.S. patent application Ser. No. 13/152,119, Jun. 2, 2011, entitled “SENSORY INPUT PROCESSING APPARATUS AND METHODS”, each incorporated herein by reference in its entirety.
A typical artificial spiking neural network, such as the network 100 shown for example in FIG. 1 herein, comprises a plurality of units (or nodes) 102, which correspond to neurons in a biological neural network. Any given unit 102 may receive input via connections 104, also referred to as communications channels, or synaptic connections. Any given unit 102 may further be connected to other units via connections 112, also referred to as communications channels, or synaptic connections. The units (e.g., the units 106 in FIG. 1) providing inputs to any given unit via for example connections 104, are commonly referred to as the pre-synaptic units, while the unit receiving the inputs (e.g., the units 102 in FIG. 1) is referred to as the post-synaptic unit. Furthermore, the post-synaptic unit of one unit layer (e.g. the units 102 in FIG. 1) can act as the pre-synaptic unit for the subsequent upper layer of units (not shown).
Each of the connections (104, 112 in FIG. 1) is assigned, inter alia, a connection efficacy (which in general refers to a magnitude and/or probability of influence of pre-synaptic spike to firing of a post-synaptic neuron, and may comprise for example a parameter such as synaptic weight, by which one or more state variables of post synaptic unit are changed). During operation of the pulse-code network (e.g., the network 100), synaptic weights are typically adjusted using a mechanism such as e.g., spike-timing dependent plasticity (STDP) in order to implement, among other things, learning by the network.
One such adaptation mechanism is illustrated with respect to FIGS. 2-3. Traces 200, 210 in FIG. 2 depict pre-synaptic input spike train (delivered for example via connection 104_1 in FIG. 1) and post synaptic output spike train (generated, for example, by the neuron 102_1 in FIG. 1), respectively.
Properties of the connections 104 (such as weights w) are typically adjusted based on relative timing between the pre-synaptic input (e.g., the pulses 202, 204, 206, 208 in FIG. 2) and post-synaptic output pulses (e.g., the pulses 214, 216, 218 in FIG. 2). One typical STDP weight adaptation rule is illustrated in FIG. 3, where the rule 300 depicts synaptic weight change Δw as a function of the time difference between the time of post-synaptic output generation and arrival of pre synaptic input Δt=tpost−tpre. In some implementations, synaptic connections (e.g., the connections 104 in FIG. 1) delivering pre-synaptic input prior to the generation of post-synaptic response are potentiated (as indicated by Δw>0 associated with the curve 302), while synaptic connections (e.g., the connections 104 in FIG. 1) delivering pre-synaptic input subsequent to the generation of post-synaptic response are depressed (as indicated by Δw<0 associated with the curve 304 in FIG. 3). By way of illustration, when the post-synaptic pulse 208 in FIG. 2 is generated: (i) connection associated with the pre-synaptic input 214 precedes the output pulse (indicated by the line denoted 224) and it is potentiated (Δw>0 in FIG. 3 and the weight is increased); and (ii) connections associated with the pre-synaptic input 216, 218 that follow are depressed (Δw<0 in FIG. 3 and the weights are decreased).
Neural networks, such as that illustrated in FIG. 1, are often utilized to process visual signals, such as spiking signals provided by retinal gangling cells (RGCs) of neuroretina and/or output interface of artificial retina. In these applications, neurons (e.g., the neurons 102 in FIG. 1) receive inputs (e.g. via connections 104) from RGCs associated with overlapping regions of the visual field, denoted by dashed curve 108 in FIG. 1. Over time, the neurons 102 learn to respond to features presented in the visual input, and develop respective receptive fields as described, for example, in co-pending and co-owned U.S. patent application Ser. No. 13/152,119, entitled “SENSORY INPUT PROCESSING APPARATUS AND METHODS”, incorporated supra. Learning may involve changing of synaptic weights of feed-forward connections to the neurons from RGCs, for example. In some cases, these learned receptive fields correspond to orientated receptive fields resembling simple cells observed in biology.
In some applications, visual receptive fields may be described in two dimensions as response to certain patterns of light representing visual features in the image (e.g. oriented lines, color patterns, or more complex shapes, etc.) Receptive fields may also have a temporal component, such that the receptive field is a two-dimensional shape that changes with time until the point of firing of the cell.
Accordingly, there is a salient need for additional mechanisms to produce receptive field diversity and improve feature detection capabilities of the network when processing sensory inputs, such as for example by introducing competition for features among neurons.